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© Computer elements in a massively parallel com- 
puter system are interconnected in such a way that 
the number of connections per element can be bal- 
anced against the network diameter or worst case 
path length. This is done by creating a topology that 
maintains topological properties of hypercubes yet 
improves flexibility by enumerating the nodes of the 
network in number systems whose base can be 
varied. Topologies are generated in which nodes are 
not always connected when their addresses differ in 
a single digit. A new variable d is introduced, the 
purpose of which is to control the overall density of 
the network by controlling the number of intermedi- 
ate arc connections within the rings of the network. 
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The present invention relates to massively par- 
allel computing systems. 

As the cost of computer elements declines, the 
possibility of building very large networks is being 
realised. Ralph Duncan in "A Survey of Parallel 
Computer Architectures", Computer, February 
1990, pp. 5 to 16, reviews alternative approaches 
to parallel processing. One class of parallel ar- 
chitecture described by Duncan involves multiple 
processors autonomously executing diverse 
instructions on diverse data, which Duncan clas- 
sifies as MIMD for multiple instruction, multiple 
data streams. Within this class of parallel architec- 
ture are the ring, mesh, tree, and hypercube 
topologies. The ring topology is characterised as 
having a communication diameter of N/2 where N 
is the number of nodes. The communication diam- 
eter is defined as the worst case path length. In 
contrast, a two-dimensional mesh topology typi- 
cally has a communication diameter of 2(N-1) 
where, again, N is the number of nodes, and this is 
also the typical communication diameter of tree 
topologies, although in both cases strategies have 
been employed to reduce the communication di- 
ameters. 

Of these known network topologies, the hyper- 
cube has some interesting topological properties. A 
Boolean n-cube or "hypercube" topology uses 
N = 2 n processors arranged in an n-dimensional ar- 
ray. Each node, or processor, in the array has 
n = log2N bidirectional links to adjacent nodes. Fig- 
ure 1 illustrates hypercubes for n = 2,3,4. Examples 
of existing hypercube machines include Caltech's 
CosmicCube, Intel's iPSC/2 and NCube's 
Ncube/10. Descriptions of these and other hyper- 
cube computer systems are provided by John P. 
Hayes et al. in "Hypercube Supercomputers", Pro- 
ceedings of the IEEE, vol. 77, no. 12, December 
1989, pp. 1829 to 1841. Ophir Fheder in 
"Multiprocessor Algorithms for Relational- Database 
Operations on Hypercube Systems", Computer, 
November 1990, pp. 13 to 28, provides a tutorial 
on hypercube systems, particularly as database 
engines. Hayes et al. list a variety of representative 
applications for hypercube computers. 

With specific reference to Figure 1, note the 
numbering convention for the addresses of the 
nodes. For n = 2, there are four nodes (N = 2 2 ) with 
addresses 00, 01, 01, and 11. Each node is con- 
nected to two other nodes, and the rule applied is 
that each node is connected to an adjacent node 
having an address which differs by only one bit. 
For n = 3, there are eight nodes (N=2 3 ) with ad- 
dresses 000, 001, 010, 011, 100, 101, 110, and 
111, but the connection scheme is the same with, 
for example, node 000 being connected to nodes 
001, 010 and 100. There are sixteen nodes for the 
case where n=4 (N = 2 4 ), but still the same con- 
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nection scheme is employed. Laxmi N. Bhuyan and 
Dharma P. Agrawal in "Generalized Hypercube and 
Hyperbus Structures for a Computer Network", 
IEEE Transactions on Computers, vol. c-33, no. 4, 
5 April 1984, pp. 323-333, describe a generalisation 
of the binary hypercube to a variable radix number- 
ing. In this scheme, the nodes of a network are 
numbered in a system in which each digit of the 
number can be based on a different radix. Nodes 
w are connected whenever the resulting node ad- 
dresses differ in exactly one digit. 

While hypercube architectures offer some very 
attractive topological properties, as these networks 
go beyond a few thousand nodes, the numbers of 
15 connections per node and path lengths grow in 
fixed ways. If either connections per node or path 
lengths become too large for a given technology, 
the designer is forced to develop new network 
algorithms from scratch. These new ad hoc 
20 topologies may then adversely effect the perfor- 
mance of existing programs that ran well on small- 
er networks with older topologies. 

According to the present invention, there is 
provided a computing system in the form of a non- 
25 binary hypercube comprising: 

a plurality of nodes, N, where N = b n , b and n 
being positive integers and b>2, each node having 
an identifying address numbered according to a 
number system base b; and 
30 means for interconnecting each node to nodes 

having addresses differing from the node's address 
by exactly one digit, the number of connections per 
node being ulog b N, where u=2d, 1 ^ d ^ b/2 if b 
is even and 1 £ d < (b-1)/2 if b is odd, except if b 
35 is even and d = b/2, in which case u = b-1 . 

The present invention enables a method of 
interconnecting processors to form a generalised 
hypercube structure to be provided that supports 
an unlimited number of nodes. 
40 In such a computer structure, the number of 

interconnections per processor can be adjusted 
against the network diameter. According to the 
invention, there is provided a method for intercon- 
necting processing elements in such a way that the 
45 number of connections per element can be bal- 
anced against the network diameter. This is done 
by creating a topology that maintains many of the 
well known and desirable topological properties of 
hypercubes while improving flexibility by enumerat- 
50 ing the nodes of the network in number systems 
whose base can be varied. When using a base two 
number system, this method creates the familiar 
Boolean binary hypercube topology. However, in 
contrast to a Boolean binary hypercube, the prac- 
55 tice of the invention results in a non-binary hyper- 
cube having fewer interconnections permitting the 
practical realisation of very large computer systems 
having an unlimited number of nodes. 
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The subject invention is a refinement of the 
scheme described by Laxrni N. Bhuyan and 
Dharma P. Agrawal, supra. The invention generates 
topologies in which nodes are not always con- 
nected when their addresses differ in a single digit. 
The invention introduces a new variable d, the 
purpose of which is to control the overall density of 
the network by controlling the number of intermedi- 
ate arc connections within the rings of the network. 
For a network having N nodes, the assumption is 
made that there exist positive integers b and n 
such that N = b n . The new variable d is picked such 
that 1 < d ^ b/2, if b is even, and 1 $ d S (b-l)/2, if 
b is odd. The nodes of the network are numbered 
to the base b. Two nodes, x and y, of the network 
if and only if 

1 . the address of x differs from the address of y 
in exactly one digit, and 

2. for the digit i where Xj is not equal to y u yj = - 
(Xj + j) mod bor yi = (Xj-j) mod b for some j 5 d. 

The invention will be better understood from 
the following detailed description of embodiments 
of the invention with reference to the drawings, in 
which: 

Figure 1 is a pictorial representation of some 
exemplary Boolean hypercube topologies; 
Figure 2 shows the topology of a typical ring in . 
a base 8 network where d = 1 ; 
Figure 3 shows the topology of a typical ring in 
a base 8 network where d = 2; 
Figure 4 shows the topology of a typical ring in 
a base 8 network where d = 3; 
Figure 5 shows the topology of a typical ring in 
a base 8 network where d = 4; 
Figure 6 is a block diagram of a 125 computer 
network where b = 5, n = 3 and d = 1 ; 
Figure 7 is a block diagram of the lowest level 
of packaging of an implementation of the inven- 
tion composed of 4096 nodes; 
Figure 8 is a block diagram of the next level of 
packaging of the embodiment shown in Figure 
7; and 

Figure 9 is a block diagram of the highest level 
of packaging of the embodiment shown in Fig- 
ure 7 of the drawings. 
Before describing a specific implementation of 
the invention, the network connection algorithm is 
discussed in general terms. The algorithm is based 
on the assumption that there exists positive in- 
tegers b and n such that N = b n , where N is the 
number of nodes in the network. An integer d is 
chosen between one and b/2, if b is even, or 
between one and (b-l)/2, if b is odd. Once the 
integer d has been selected, the nodes of the 
network are numbered to the base b. In other 
words, 

N = a m b m + a m .ib m ' 1 + . . . + aib 1 + aob°, 


where ao, ai, . . ., am.!, a m are non-negative in- 
tegers each less than b. For a base ten (b= 10) 
number system, the symbols aj are called "digits", 

5 and for a base two (b = 2) number system, the 
symbols a\ are called "bits". Without loss of gen- 
erality, the symbols a; are referred to hereinafter as 
"digits" even though the base number system may 
be other than ten. 

io Next, connect two nodes, x and y, of the net- 

work if and only if 

1 . the address of x differs from the address of y 
in exactly one digit, and 

2. for the digit i where X| is not equal to y>, y» = - 
75 (Xi + j) mod b or Vj = (x r j) mod b for some j < d. 

Notice that -x mod b = (y-x) mod b. For example, 
if b = 5 and d = 1 , then a node whose address is 
1234 would be connected to nodes having ad- 
dresses 2234, 0234, 1334, 1134, 1224, 1244, 1233, 

20 and 1230. Notice that d controls the density of the 
network. The larger d becomes, the denser the 
network becomes. 

During the application of the algorithm, the 
ports of the various nodes are labelled to indicate 

25 both the digit and digit value of the node to which 
they connect. For example, a port labelled 3,7 
would connect to a node whose third digit is 7. The 
number of connections per node is u.log b N, where 
u = 2d, except if b is even and d = b/2 in which 

30 case U = b-1. This can be seen to be true by 
considering that for each integer less than or equal 
to d, there are two connections for each digit of a 
node address. The number of digits in the node 
address is log b . The diameter of the network is 

35 given by 

b-l(log L N) t 


where 


is always rounded up to the next integer when it is 

so not an integer. 

Referring now to the drawings, and more par- 
ticularly to Figure 2, there is shown the topology of 
a typical ring in a base 8 (i.e., b = 8) network where 
d = 1. There are eight computers or nodes, N, in 

55 this network so that n = 1 or N-8 1 . While this 
illustration is trivial, the next three figures illustrate 
the effect of increasing the value of d on the 
density of the network. Figure 3 shows the case for 
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d = 2. In this case, each computer connects to four 
other computers. Figure 4 shows the case for d = 3. 
In this case, each computer connects to six other 
computers. Figure 5 shows the case for d = 4. In 
this case each computer connects to seven other 
computers. Thus, Figures 2, 3, 4, and 5 provide a 
graphic illustration of the increasing density of a 
simple eight node network for an increasing value 
of d. 

A more practical application of the interconnec- 
tion technique according to the invention is illus- 
trated in Figure 6 which shows a 125 computer 
network. In this example, the base, b, is set to 
equal five so that n = 3; that is, N = b n = 5 3 = 125. In 
order to minimise the density of this network, d is 
set equal to one. This network is implemented in 
five planes 10, 11, 12, 13, and 14, with each plane 
having twenty-five computers 15. Within each plane 
there are five rings and five loops. Each computer 
in the network is connected to six other computers, 
two computers within its ring, two computers within 
its loop and two computers in different planes. 

Even for this still relatively simple network, the 
selection of the variable d allows a control of the 
density of the network not known in the prior art. 
The network shown in Figure 6 may be easily 
realised as a practical embodiment of five printed 
circuit boards with edge connectors. However, the 
invention is especially useful for very large net- 
works where interconnection densities become 
critical to whether an implementation is realisable 
in the real world. 

To illustrate this, a specific implementation of 
the invention for d = 1 , b = 8 and n = 4 is shown in 
Figures 7, 8 and 9. Thus, for this implementation 
N = b n = 8 A =4096. For this very large number of 
nodes, the density variable d is chosen to be one 
to minimise the density. 

Figure 7 shows the lowest level of packaging. 
Each box 20o to 2O7 contains eight computers 
interconnected with forty-eight (3x16) data paths 
22i , 22 2 and 22 3 per box. The complete package 
24 in Figure 7 thus contains sixty-four (8x8) com- 
puters. The package 24 is interconnected to the 
rest of the computers in the system with 256 
(2x128) data paths 26i and 26 2 . 

Figure 8 shows the next level of packaging. 
Each package 24o to 247 contains 64 computers 
with 256 data paths as shown in Figure 7. The 
larger package 30 thus contains 512 (8x64) com- 
puters supporting 1024 (2x512) data paths. Figure 
9 shows the highest level of packaging containing 
eight packages 30 0 to 307, each containing 512 
computers and 1024 data paths. There are 4096 
(8x512) computers with 1024 data paths in highest 
level of packaging. 

In this network of 4096 nodes, each node is 
directly connected only eight other nodes. This 
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permits any node to communicate with any other 
node in the network by involving no more than 
sixteen intermediate nodes. This compares well 
with the case of the Boolean binary hypercube 

5 which demands a denser network with each node 
connected to twelve other nodes rather than eight. 
Thus, the invention allows for a reduction of 33% in 
wiring density as compared with the Boolean bi- 
nary hypercube. This reduction, however, is traded 

10 for a concomitant 33% increase in network diam- 
eter. 

To route messages in a network according to 
the invention, first compare the destination address 
with the current address. If they are the same, 

75 routing has been completed. If not, select some 
digit, i, for which the current and destination ad- 
dresses are different. Forward the message out of 
the port for this digit that differs from the destina- 
tion address by the least amount. Continue this 

20 procedure until the current and destination ad- 
dresses are equal. Note that the selection of the 
digit to work on at a particular time can be based 
on which ports are not busy or damaged. This is 
common procedure for hypercubes. A port is said 

25 to be damaged if the port itself is damaged or the 
port to which it connects is damaged. 

The invention simplifies the interconnection of 
massively parallel computer systems yet preserves 
the desirable properties of a hypercube. These 

30 properties are (1) a large number of alternate 
paths, (2) very high aggregate bandwidth, and (3) 
uniform connections. Existing methods can be 
used to used to route messages within the network. 
Generally, the invention provides a non-binary 

35 hypercube with less density thereby allowing the 
practical realisation of massively parallel computer 
systems having an unlimited number of nodes, but 
as illustrated in Figure 6, the invention also has 
practical application for smaller computer networks. 

40 Computer elements in a massively parallel 

computer system are interconnected in such a way 
that the number of connections per element can be 
balanced against the network diameter or worst 
case path length. This is done by creating a topol- 

45 ogy that maintains topological properties of hyper- 
cubes yet improves flexibility by enumerating the 
nodes of the network in number systems whose 
base can be varied. Topologies are generated in 
which nodes are not always connected when their 

50 addresses differ in a single digit. A new variable d 
is introduced, the purpose of which is to control the 
overall density of the network by controlling the 
number of intermediate arc connections within the 
rings of the network. 

55 While the invention has been described in 

terms of illustrative and preferred embodiments, 
those skilled in the art will recognise that the inven- 
tion can be practised with modification within the 
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scope of the appended claims. 
Claims 

1. A computing system topological^ arranged to 5 
form a non-binary hypercube comprising: 

a plurality of nodes, N, where N = b n , b and 
n being positive integers and b>2, each node 
having an identifying address numbered ac- 
cording to a number system base b; and 10 

means for interconnecting the nodes so 
that each node is connected to nodes having 
addresses differing from the node's address at 
exactly one digit, the number of connections 
per node being ulog b N, where u=2d, 1 £ d £ 15 
b/2 if b is even and 1 £ d £ (b-1)/2 if b is odd, 
except if b is even and d = b/2,' in which case 
u = b-1 . 

2. A computing system as claimed in Claim 1 20 
wherein two nodes, x and y, of the network are 
connected if and only if 

1 . the address of x differs from the address 
of y in exactly one digit, and 

2. for the digit i where x* is not equal to yi, 25 
yi = (Xj+j) mod b or yj = (xrj) mod b for 
some j £ d. 

3. A computing .system as claimed in Claim 1 or 
Claim 2 wherein b = 8, n = 1 , and d = 2. 30 

4. A computing system as claimed in Claim 1 or 
Claim 2 wherein b = 8, n = 1 , and d = 3. 

5. A computing system as claimed in Claim 1 or 35 
Claim 2 wherein b = 8, n = 1 , and d = 4. 

6. A computing system as claimed in Claim 1 or 
Claim 2 wherein b = 5, n = 3, and d = 1 . 

40 

7. A computing system as claimed in Claim 1 or 
Claim 2 wherein b = 8, n = 4, and d = 1 . 

8. A method of interconnecting a plurality of 
nodes in a network comprising the steps of: 45 

defining b and n such that the number N 
of nodes equals b" and b and n are positive 
integers and b>2; 

choosing a positive integer d such that 1 £ 
d < b/2 if b is even and 1 £ d £ (b-1)/2 if b is 50 
odd; 

addressing each of the nodes by number- 
ing them using numbers represented by a 
base b numbering system; and 

connecting two nodes if and only if the 55 
address of a first one of said two computing 
elements differs from the address of the sec- 
ond one of said two nodes in exactly one digit, 

5 
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the i 1h digit and, for the unique i th digit of the 
addresses of said two computing elements, 
symbolised by Xj and yj, respectively, where Xj 
is not equal to yi, yi = (xj+j) mod b or y, - (xpj) 
mod b for some j £ d. 
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© Computer elements in a massively parallel com- 
puter system are interconnected in such a way that 
the number of connections per element can be bal- 
anced against the network diameter or worst case 
path length. This is done by creating a topology that 
maintains topological properties of hypercubes yet 
improves flexibility by enumerating the nodes of the 
network in number systems whose base can be 
varied. Topologies are generated in which nodes are 
not always connected when their addresses differ in 
a single digit. A new variable d is introduced, the 
purpose of which is to control the overall density of 
the network by controlling the number of intermedi- 
ate arc connections within the rings of the network. 
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